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In today’s world, much of what used to be individual work has 
become collaborative. Moreover, complex change initiatives often 
require individuals within and across organizations to team up to set 
and achieve meaningful goals. Our role as researchers and evaluators 
is to offer support that can be used to strengthen the work of 
organizations and interdisciplinary teams. We focus on four broad 
categories: 

1. identifying existing evidence in the field that can be used to 
inform the quality of a project, 

2. gathering information from stakeholders to identify key needs in 
the field, 

3. conducting formative and summative evaluation for programs 
and initiatives, and 

4. offering technical assistance support regarding how to use data. 

Our work with pepnet 2, in particular, has afforded us several lessons regarding 
the significance of evaluative thinking to collaborative endeavors. Evaluative 
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thinking—a process that involves 
systematic results-oriented reasoning—is 
essential for collaborations to be successful. 
Evaluative thinking focuses individuals 
and their discussion on the outcomes that 
are expected from an endeavor, how those 
outcomes might be achieved, the research 
and/or evidence that informs results, and 
ongoing examination and reflection on 
data regarding the progress of the 
collaboration. (Patton, 2014). 

When infused into the culture and 
activities of a collaboration, evaluative 
thinking propels individuals forward in 
the same direction and increases the 
likelihood of success down the road. 

Lesson 1— Collaboration Alone 
is Insufficient 

With terms like cross-sectoral, public-private 
partnership, professional learning community, 
social network, interdisciplinary, multi¬ 
disciplinary, alliance, consortium, and 


collaborative increasingly widespread, it 
may be tempting to view endeavors 
undertaken through group cooperation as a 
panacea, the single way to effect complex 
change. Certainly, many of the persistent 
and intractable social problems are more 
likely to be solved by the collective action 
of key individuals with diverse 
perspectives. 

Collaboration, however, is a means to an 
end, not the end itself. Simply bringing 
together people with common interests to 
work on a challenging initiative is not a 
failsafe approach. In fact, it is insufficient. 
Collaboration is more likely to be fruitful 
when team members are able to exercise 
evaluative thinking systematically about 
the change they want to achieve and 
properly ground efforts in evidence. 

In our work, we have seen the benefits of 
evaluative thinking as a catalyst for 
progress towards complex problem solving. 
As teams begin to think analytically 
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Figure I: Pepnet 2 Theory of Change 


Below is an overview of the theory of change that underlies the work of pepnet 2. 
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through their purpose and arrive at 
common goals, they generate 
momentum, enthusiasm, and greater 
commitment to work. Moreover, when 
teams have a plan that is rooted in 
evidence and includes indicators to 
monitor progress, they can begin the 
meaningful work that is required for 
change. By contrast, when teams fail to 
craft a coherent plan for moving forward 
that is grounded in systematic, results- 
oriented evaluative thinking, they often 
struggle to move beyond convening and 
toward collective action. 

Lesson 2— Begin with a 
Theory of Change 

A theory of change provides a useful 
starting point for effective collaboration. 
It explains the principles underlying an 
initiative and outlines how desired 
outcomes will be produced. This theory 
can come in various shapes and sizes, but 
it generally has three main components 


(Annie E. Casey Foundation, 2004): 

1. Long-term and short-term 
outcomes—The long-term goal for 
pepnet 2 is to improve access, 
opportunity, and postsecondary 
success for students who are deaf or 
hard of hearing. In order for these 
long-term changes to occur, short¬ 
term changes requiring new 
evidence-based practices, policies, 
and programs are required. (See 
Figure 1.) 

2. Strategies—In order to accomplish 
these short-term outcomes, pepnet 
2 engages in several strategies and 
activities, including technical 
assistance, training, research, and 
convening. 

3. Explicit assumptions— 
Assumptions about why strategies 
will work should be explicit. The 
assumption underlying the work of 


pepnet 2 is that numerous needs in 
the field should be addressed in 
order to achieve greater 
postsecondary success for deaf and 
hard of hearing individuals, 
including stronger evidence-based 
knowledge and tools, greater 
cultural competence, better 
transition and advocacy, and 
integrated services. 

By focusing on select strategies that 
are driven by needs in the field, our 
assumption is that short-term outcomes 
(i.e., greater knowledge regarding 
effective transition) will lead to long¬ 
term outcomes (i.e., deaf students' 
success in the postsecondary 
environment). 

Stakeholders who are engaged in 
collaborative change efforts often can 
identify the various strategies and 
activities that they hope to implement. 
However, without a theory of change, 
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the desired outcomes and assumptions 
underlying those strategies are not made 
explicit, and the connection between 
strategies and outcomes remains unclear. 
Taking the time to generate a theory of 
change ensures a chain of reasoning 
grounding the new initiative and 
promotes a shared understanding of how 
to move forward. Moreover, the theory 
can be used to explain to others what the 
collaboration is about and provide the 
foundation to determine how to measure 
its progress. 

When crafting a theory of change, it is 
often useful to begin with all 
stakeholders at the table and address the 
following questions: What is the group’s 
desired long-term change? What will look 
different after the group’s strategies have been 
implemented? After addressing these 
questions, teams should work to identify 
the short-term changes required to 
accomplish the long-term goal. The 
Annie E. Casey Foundation (2004), an 
organization dedicated to helping 
disadvantaged children in the United 
States, provides a practical guide for 
structuring theories of change and 
suggests three main types of outcomes: 

Taking the time to 
generate a theory of 
change ensures a 
chain of reasoning 
grounding the new 
initiative and 
promotes a shared 
understanding of how 
to move forward. 


1. Impact—Changes in people’s lives, 
including knowledge, skills, 
behaviors, health, or conditions for 
communities. 

2. Influence—Changes in 
institutions, service systems, 
community norms, partnerships, 
policies, or regulations. 

3. Leverage—Changes in public or 
private funding and resource 
allocation. 

Once the short- and long-term 
outcomes are mapped out, strategies and 
activities can be generated that are likely 
to lead to the change envisioned. As 


strategies are clarified and connected to 
outcomes, a chain of logical reasoning 
should begin to emerge. (See Figure 2.) 

Lesson 3— Ground 
Discussion in Research and 
Evidence 

Ideally, a theory of change includes a 
clear articulation of assumptions about 
why the proposed activities will lead to 
the desired short- and long-term 
outcomes. Without a critical look at the 
available research and evidence base, 
decisions about planned strategies and 
activities may be based on intuition and 
individual experiences or even on 
political demands. Furthermore, 


Figure 2: Chain of Logical Reasoning 

Below is an example of a chain of logical reasoning for a potential 
outcome focused on improving transition. 


Long-term outcome: Improved postsecondary outcomes for deaf and hard of 

hearing students 
as a result of: 

4 

Outcome 4: Stronger transition plans for shidents that are aligned with 
students’ longer-term educational and career goals 
as a result of: 

4 

Outcome 3: Teachers implementing more student-led IEPs 
as a result of: 


Outcome 2: Increased confidence to incorporate student-led IEPs 
as a result of: 

4 

Outcome 1: Greater knowledge and skills for teachers 
as a result of: 

4 

Strategy: Train teachers about effective practices for implementing student-led 
Individualized Education Programs (IEPs) 
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collaborative teams may be 
drawing individual 
knowledge from different 
perspectives, with resulting 
divergent assumptions 
about why particular 
strategies may or may not 
be effective. Looking to the 
available research and 
evidence is thus an 
important component in 
developing a shared 
understanding of the 
rationale for choosing 
activities. 

The first step to 
reviewing the existing 
literature is to identify the 
evidence that supports the needs of the 
project. There are many different kinds 
of evidence that may be integrated into a 
rationale that explains why proposed 
strategies and activities will lead to 
anticipated short- and long-term 
outcomes. When developing pepnet 2’s 
theory of change, our goal was to 
provide both theoretical perspectives 
and data. We drew from theoretical 
perspectives in the fields of human 
development, cultural psychology, and 
deaf education. We asked what factors 
were important in understanding 
potential barriers or supports for deaf 
and hard of hearing individuals in 
achieving their educational, work, and 
personal goals. Data and empirical 
evidence were important. In our pepnet 
2 work, we synthesized existing 
literature on key topics, such as the 
effectiveness of accommodations, and we 
gathered information from the field to 
answer questions that the extant research 
literature could not provide. 

From a theoretical perspective, the 
pepnet 2 Research and Evidence 
Synthesis team felt it was important to 
include cultural competency frameworks 
in considering access and options for 
individuals who are deaf or hard of 
hearing. This perspective focuses 
attention on building the skills and 
attitudes of professionals that serve 
individuals who are deaf or hard of 


Ongoing reflection 
is critical. This 
allows collaborators 
to refine strategies 
and to measure 
outcomes to 
determine 
effectiveness. 

hearing, and cultural competency is a 
part of creating an open and positive 
learning or work environment. Synthesis 
of current data was important in laying 
the foundation for decisions about future 
activities or programs. More specifically, 
we drew upon the current demographic 
data available about current high school 
completion, college enrollment, 
postsecondary persistence, and 
employment for individuals who are deaf 
or hard of hearing. Significant 
demographic shifts over the last 10 years 
have occurred, and updating was 
essential. 


A second example of where 
evidence played a role in the 
development of the theory of 
change was in the 
articulation of the potential 
short-term outcomes that 
would result from the 
proposed activities. For 
example, research shows that 
self-determination is 
predictive of stronger 
postsecondary outcomes for 
students with disabilities 
(Konrad & Test, 2004). 
Although this research was 
not conducted with deaf or 
hard of hearing students, 
pepnet 2 made the 
assumption that fostering students to 
lead IEPs might have similar results for 
our students. When requested, pepnet 
2’s Research and Evidence Synthesis 
team provided this kind of information 
to collaborative teams throughout the 
organization. 

Lesson 4— Monitor the 
Progress 

Ongoing reflection is critical. This 
allows collaborators to refine strategies 
and to measure outcomes to determine 
effectiveness. A well-defined theory of 
change should suggest key indicators 
that can be used for monitoring and 
evaluating the collaboration. 

For example, if teachers are trained to 
effectively implement student-led IEPs, 
then it follows that the two primary 
indicators of progress might be the 
number of trainings conducted and the 
number of teachers trained. If there is an 
insufficient number of trainings or if 
attendance at trainings is low, it is 
unlikely that the changes articulated in 
the theory would occur. However, if 
those indicators were measured 
frequently and reviewed, stakeholders 
could examine the data and intervene 
appropriately before the end of the 
initiative. 

It is not sufficient to only examine 
data related to the strategies and 
activities in an initiative; an 
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examination of the outcomes is also 
warranted to determine whether the 
collaborative endeavor has produced the 
intended changes. Potential indicators in 
the example above might include pre- 
and post-measures of the following 
outcomes: knowledge about and 
confidence in the IEP process of students 
and teachers, the number of student-led 
IEPs that are taking place, and the 
number of IEPs with transition plans 
that are aligned to students’ longer-term 
educational and career goals. 

Putting It Together 

Taken together, these lessons illustrate 
how transparency in planning activities, 
naming assumptions behind their 
effectiveness, and monitoring outcomes 
can assist collaborative teams in the 
development and implementation of 
activities. Preparing individuals who are 
deaf or hard of hearing for future 
education and work opportunities often 
involves the collaboration of people or 


agencies with multiple perspectives, 
diverse training, and a broad range of 
experiences. A collaborative model in 
effective program and service 
development therefore requires an 
anchor that represents the shared beliefs 
and assumptions about what is being 
done, why, and to what end. This is the 


purpose of devising or adopting an 
underlying theory of change and 
combining it with evaluative thinking 
for the collaboration. Our work with 
pepnet 2 represents an evolving model 
that can provide a platform for those 
seeking to engage in meaningful 
discussion and collaborative activities. 
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